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Abstract 

We study irreducible time-homogenous Markov chains with finite 
state space in discrete time. We obtain results on the sensitivity of the 
stationary distribution and other statistical quantities with respect to 
perturbations of the transition matrix. We define a new closeness rela- 
tion between transition matrices, and use graph-theoretic techniques, 
in contrast with the matrix analysis techniques previously used. 
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1 Introduction 



The present paper concerns irreducible time-homogenous Markov chains 
with a finite state space in discrete time. We are interested in the effects of 
perturbations of the transition matrix on the stationary distribution and on 
other statistical quantities. 

This question has a long history, starting with Schweizer [?]. Let S be 
a finite set, and q = (q(t \ s)) S; teS an irreducible transition matrix. Let 
q = ((q(t | s)) St teS be another transition matrix. Denote by fx = (fi s ) se s 
and fl = (/2 s ) sG 5 the stationary distributions that correspond to q and q 
respectively. Schweizer [?] estimated — At||oo as a function of ||g — g||ooi 
using the fundamental matrix of Kemeny and Snell [?]. A vast literature 
has followed up Schweizer, and provided various estimates to — j£||oo and 
(s G S) as a function of ||g — g||oo- See, e.g., [?], [?], [?], [?], [?], [?]. 
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O'Cinneide [?] studied the effects of entry- wise relative perturbations on 



Ms -Ms 

Ms 



as a 



the stationary distribution; that is, he provided a bound on 
function of max s , e5 {|(|),||)}. 

Our paper differs from the existing literature in three respects. First, 
we introduce a new way to measure the difference between two transition 
matrices. In the spirit of the entry-wise relative perturbations of O'Cinneide 
[?] , our measure is max || 1 — | j , but the maximum is not taken over 
all pairs s,t € S. Rather, the maximum is taken over all frequent transitions 
s — > t. Formally, the notion of closeness between q and q is the following. 
For a transition matrix q, we define ( q = min0 CCC 5 J2 S £C Ms?(C I s )> where 
C = S \ C is the complement of C in S. This is a variant of the conductance 
(see, e.g., [?], [?], [?]). Given e,(3 > 0, we say that q is (e, /3)-close to q if 



for every two states s,t G S, 



i _ q(t\ s ) 

1 q(t\8) 



< (3 whenever (a) fj, s q(t\s) > e( q or 



(b) /j, s q(t\s) > eC, q . Condition (a) holds whenever the transition from s to 
t occurs frequently. Condition (b) is not analogous to (a), since it involves 
the stationary distribution of q, and the transition matrix q. 

Provided e and (3 are small enough, we show that q is irreducible, and 

is close to 1 for each s€S. 
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The logic behind this closeness relation, and its novelty, is that even 
large perturbations that occur in rarely visited states should not affect too 
much the stationary distribution. This point is illustrated in the example 
that is studied in the next section. 

A motivation for using this closeness measure is the following statistical 
implication. Assume a statistician observes a realization si,...,sn of (the 
first N components of) a Markov chain with unknown irreducible transition 
matrix q. Given these observations, he computes the empirical transition 
matrix q and the invariant distribution ju of q (if Sjv = si, coincides with 
the empirical occupancy measure). For fixed e,/3 > 0, if N is large enough, 
q is (e, /?)-close to q with probability close to one, hence /I is an accurate 
estimate of the invariant distribution of the underlying Markov chain. 

Second, instead of using matrix analysis, we use the graph techniques 
developed by Freidlin and Wenzell [?] and extensively used in the analysis 
of Markov chains with rare transitions (see, e.g., Catoni [?]). 

Third, our approach allows us to estimate the sensitivity of other sta- 
tistical quantities, such as the exit distribution from a given set and the 
average length of visits to a given set. 

The paper is organized as follows. Section [21 contains the statements 
of the main results. We also study there an example that illustrates the 
advantage of our closeness relation, and compares the bound we derive to 
existing bounds. Section |3] briefly recalls standard formulas, and states few 
elementary properties of graphs. Section 0] is devoted to the proof of the 
main result. The last section deals with a variation of the main result. 

2 Notations and results 

Let S be a finite set, fixed through the paper, with at least two elements. 
For every subset C C S, C = S \ C is the complement of C in S, \C\ is 
its cardinality, and A(C) is the set of probability distributions over C. For 
s £ C C S, we write C \s instead of the more cumbersome C \ {s}. 
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2.1 Main result 

Let q be an irreducible transition matrix over S, with stationary distribution 
[j, = (fi s ) s( zs- For every C C S we denote jitc = Ssgc/^s- Let g be another 
transition matrix over S. Assuming q is irreducible, we wish to bound the 
distance between (i and jl. 

Our notion of closeness of q to q involves a measure of how mixing q is. 
Our measure involves the quantity 

C?= min I *), (1) 

which is a variant of the conductance, see e.g. [?], [?], [?]. Given C C S, 
the quantity XLgc ^s<l{C \ s) measures the average frequency of transitions 
out of C. Hence, ( q , being the lowest such frequency, is a measure of how 
isolated a subset C may be. Formally, 

Definition 1 Let e, (3 > 0. We say that a transition matrix q is (e, (5)- 



close to q if for every two states s,t £ S, 
Vsq{t\s) > s( q or (b) n s q(t\s) > e( g . 



Mil 

q(t\s) 



< whenever (a) 



Note that this closeness notion is not symmetric, since we use only the 
stationary distribution of q. 

Denote L = Yln^ ('« ) n ' 5 ''- We now state our main result. 

Theorem 2 Let [5 € (0, 1/2^ ) and let e G (0, x~^pO- For 

every irreducible 

transition matrix q on S and every transition matrix q that is (e, (3) -close to 



1. q is irreducible. 

2. Its stationary distribution fi satisfies 



1 _ Us. 

Ms 



< IS(5L for each se S. 



2.2 An example and comparison with existing bounds 

As mentioned in the introduction, many authors provided bounds for the 
sensitivity of the stationary distribution. 
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We now study an example that first highlights the logic behind the close- 
ness relation, and second shows that in some cases, the bound we give is 
better than existing bounds. 

Fix 5 € (0, 1/2). Take a Markov chain with three states and transition 
matrix as follows. 

q{2 | 1) = 1 - S, q(3 | 1) = 6, q(l | 2) = q(l | 3) = 1. 

Thus, in every other stage the process visits state 1. In particular, the 
stationary distribution /x is given by 

1 1-5 8 

Ml = ^ M = ^ - 2- 

The quantity Cq is given by 

. fl 1 -5 5 1 ,1 rN 1 -S 5) 5 

C " = mm l2'^'2'2 X5 '2 X(1 - 5) '^ + 2/ = 2- 

Fix < (3 < 1/8 = 1/2 3 , < e < /3(1 - 0)/3 A L and < -q < e. In 
particular, one can choose r\ = f3/C, where C > 1 is some fixed scalar, 
independent of 8. 

Define a transition matrix q by 

q(2 | 1) = 1 - 5, q(3 | 1) = 6, q(l | 2) = 1, q(l | 3) = 1 - r/, q(2 \3)=r,. 

Thus, we only change the transitions out of state 3. Moreover, whereas the 
relative size of the transition 3 — > 1 changed moderately, the relative size of 
the transition 3^2 changed dramatically. 



We first argue that (?is (e, /3)-close to q. Indeed, 



whenever 



s ^ 3, and 
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r] < (3. The claim now follows, since ^q{2 | 3) = 



and nzq{2 \ 3) = | x r) < ( q x e. 
Thus, Theorem |21 states that 



< 18L(3 = \8LCrj. 



This example highlights the logic behind our closeness relation. Since 
state 3 is rarely visited, the stationary distribution is not too sensitive to 
changes in transitions out of this state, even though these changes are rela- 
tively large. 
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Cho and Meyer [?] bound the sensitivity of the stationary distribution 
by the mean first passage time: 
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< — — ( LS2. x maxM+ 
2 t^s 



(2) 



where Mt s is the mean first passage time from state t to state s. 

Observe that in the example M2 t 3 = 2/5 while Mi 3 = 2/5 — 1, hence the 
bound provided in [?] for f13 ^ 3 is r] x ^, which is worse than our bound 
when 5 is small. 

In the case of two-state chains, the bound (J2J) is very close to the bound 
that can be derived from Theorem 12 up to a universal constant. Indeed, 
(J2J) then reduces to 



Ms - Ms 
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< I m a X {|g(2|l)-g(2|l)|,|g(l|2)-?(l|2)|}xma X |-^,-^y 

(3) 

On the other hand, Cq = mm {Mi(2 l (2|l)>M2<2 , (l|2)}. Hence, Theorem |2] yields 



Ms - Ms 
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< 36/3, with P = max 



g(2|l)-?(2|l) 



?(2|1) 



? (l|2)-g(l|2) 



?(1|2) 



provided /3 < 1/4 - an inequality which is slightly more precise than (J5J , up 
to a constant 72. 



Kirkland et al [?] bound the relative sensitivity by 
Ms - Ms 



Ms 



< 1 llizllk x || Ar i| 
- 2 2 11 s 1 



where ^4 is the fundamental matrix of [?], and A s is the (n — 1) x (n — 1) 
submatrix of A obtained by deleting the s'th row and column. 

1 -1 + 5 
-1 1 



One can verify that in our example A3 



A' 1 



1/5 -1 + 1/5 



. Thus, the bound for 

1/5 1/5 
worse than our bound when 5 is small. 



, so that 
is 77/25, which is 



Note that the entry-wise ratio bound given by O'Cinneide [?] is not 
useful in this example, since q(2 | 3) = while q{2 \ 3) > 0. 



6 



2.3 The transition matrix changes in a subset of states 

In some cases, the transition matrix is perturbed only in a subset S± C S, 
and restricted to Si the transition matrix is sufficiently mixing, in the sense 
that the probability to reach any state in S\ before leaving S\ is bounded 
from below. In this case, instead of taking the conductance in the definition 
of the closeness relation, one can take another quantity, which is, in a sense, 
the conductance restricted to Si. 

Such a case occurs, for example, if the state space can be partitioned 
into some subsets, the transition matrix is mixing in each subset while the 
probability to move from one subset to another is small, and one perturbs 
the transition matrix only in one of the subsets. 

Let Si be a subset of S, with > 1. Define 

sec 

Let (s n ) be a Markov chain with transition matrix q. We denote by P Sj(J 
the law of (s n ) when the initial state is s, and by E Sj(? the corresponding 
expectation. 

For every proper subset C of S we let Tq = min {n > 0, s n G C} denote 
the first hitting time of C and Tq = min {n > 1, s n € C} the first return to 
C . By convention, the minimum over an empty set is +oo. 

Definition 3 Let e, (3 > 0. We say that a transition matrix q is (e, /3)-close 



to q on Si if for every two states s,t € S, 
H s q(t\s) > eQ or (b) fi s q(t\s) > e^. 



1 _ <?(<|s) 



< (3 whenever (a) 



We now state the Theorem that corresponds to Theorem |^1 

Theorem 4 Let P £ (0, l/2l s l), a > and e G (0, \ (f x ^M). Let 
q be an irreducible transition matrix such that P, „(T^ , . = Tj,) > a for 
every s,t € Si- Then, for every transition matrix q that is (e, (3) -close to q 
on Si and that coincides with q on S\Si, we have 

1. All states of Si belong to the same recurrent set R for q. 
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2. The stationary distribution ju ofqonR satisfies 

< 18/3L, for each s G S u (4) 
where fi(s \ Si) = n s /ns 1 - 

Note that the claims in Theorem |I] differ from those in Theorem |2 It is 
no longer claimed that qis irreducible, nor that the unconditional stationary 
distributions \i and /2 are close. The statements in Theorem |I] are optimal 
in this respect. This is due to the fact that the quantity contains no 
information on the frequency of transitions out of S\. To emphasize this 
point, consider the following example. 

Assume that S = {a,b,c} and Si = {a, b}. Let e,/3 € (0, 1/2) be given. 
Let two additional parameters A and 77 be given in (0, 1), and define q as 
follows. From state a (resp. b) a chain with transition matrix q moves to c 
with probability rj, and otherwise to b (resp. to a). From state c, the chain 
remains in c with probability 1 — A, and otherwise moves to a or b with equal 
probability ^A. 

Plainly, q is irreducible, and the value of fj, a = fib depends on the ratio 
A/r/: this common value may be arbitrary close to (resp. to 1/2) provided 
A/77 is close enough to (resp. to +00). Note that (^ = fi a q({b,c} \a) = fj, a . 
Let now qbe defined exactly as q, except that the parameter 77 is replaced by 
another parameter rj £ [0,1]. As soon as 77, rj < min{e, /?}, q is (e,/3)-close 
to q. This is in particular the case if rj = 0, in which case q fails to be 
irreducible. On the other hand, even if 77 > 0, the values of 77, rj and A can 
be chosen in such a way that the inequalities 77, 77 < minje,/?} are satisfied, 
and 77 <C A <C 77. Hence, even if q is irreducible, its unconditional stationary 
distribution jl may be arbitrarily far from /j. 

2.4 Sensitivity of other quantities 

Our graph-theoretic approach allows us to obtain information on other quan- 
tities of interest. We here present the statements of the corresponding re- 



V-{s\Si) 
fjt(s\Si) 
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suits. 

We let Q S)9 (-|C) denote the law of the exit state from C: Q Stq (t\C) = 
P Sj9 (% = T t ) for t i C. Next, we set 

y^.+r-n Ut(j(s\t) 

vc ( s ) ■- ^tec !^ ; for C C S and sGC, and (5) 

l^tac ^tq[C\t) 

K c ■= Y, "c(*)E. lff [eo] for CcS. 
sec 

The numerator (resp. the denominator) in (JSJ) is the long run frequency of 
transitions from C to s (resp. from C to C). Thus, ^(s) is the probability 
that the first stage in C the process visits is s, while Kq is the average 
length of a visit to C. 

Assuming q is irreducible, the corresponding quantities for q will be 
denoted by Q Sj q, vc{s) and Kc- We now state the results on Q s s and 
that hold in the framework of Theorems |2] and 0] respectively. 

Theorem 5 Set c = 2 \S\ 2 . Under the assumptions of Theorem^ the fol- 
lowing holds: for each C C S, 

1. ||Q Si9 (. | C) - Q Si? (- | C) || < 12(3L for every s G C 

2. \K C <K C <cK C - 

Theorem 6 Set c = 2 |5| 2 . Under the assumptions of Theorem^ the fol- 
lowing holds: for each C C Si, 

1. ||Q S ,,(- | C) - Q Sj g(- | C?)|| < 12/3L for every s G C 

2. \K C <K C <cK c 

3. \K Sl < K Sl < cK Sl or K Sl ,K Sl >^s\X^-. 

We let q be an irreducible transition matrix over S. It is fixed throughout 
the paper. 
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3 Preliminaries 

Our computations are based on formulas due to Freidlin and Wenzell [?], 
that express stationary distribution, exit distributions and expected hitting 
times in graph-theoretic terms. For a discussion of some applications, we re- 
fer to Catoni [?] . These tools have also been used in the context of stochastic 
games in [?] and [?]. 

The weight of a graph is obtained from the transition probabilities cor- 
responding to the different edges of the graph. We recall these formulas 
in section 13.11 Next, we compare the weights of a given graph under a 
transition matrix gthat is close to q. 

3.1 Reminder 

Given C C S, a C -graph is a directed graph without cycle g over S such 
that: 1 

• For s £ C, there is exactly one edge starting at s, denoted by (s, g(s)). 

• For s £ C, there is no edge starting at s. 

Thus, given s € C, there is a unique path starting at s and ending at 
some t 6 C. We say that s leads to t along g. We denote by G(C) the set 
of C-graphs; for s £ C, t £ C, G s ,t(C) is the subset of graphs g £ G{C) 
such that s leads to t along g. Note that G(C) depends only on C, and not 
on the transition matrix. Note also that L bounds the number of graphs: 
L>E,cCcs\G(C)\. 

We identify each C-graph g with the collection of its edges: g = U se c{(s, g{ 

Given DCC, and g € G(C), the restriction of g to D is defined to be 
the subgraph of g that contains exactly those edges of g that start in D. 
Thus, it is the D-graph g' = U se u {(5,5(5))}. 

For every g G G(C), we define the weight of g under q by 

v{g)-= n ^k)- 

Our C-graphs correspond to C-graphs in [?], [?]. 
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The following formulas were derived by Freidlin and Wentzell [?], Lem- 
mas 6.3.1, 6.3.4 and 6.3.3. For more direct statements and alternative proofs 
see Catoni [?]. 

Proposition 7 (Freidlin- Wenzell, 1984) Let (S, q) be a Markov chain. 
• If q is irreducible then for every s £ S 

^ = E G (g\w)P(fl) (6) 

E^s Eg(S\{!/}) p(9 ) 



For every proper subset C of S and every s € C , 

r n T,g(c\{s})p(9) + Etec,t#s EG s ,,(c\{t})P(g) , , 
E ^ [%] = T^W) ' (7) 

Q s , q (t\C) = ^G(G)P(9) ^ ^ t ^ c (g) 
Z^G s ,t(C)PK9) 

3.2 Basic properties 

In this section we provide basic properties of weights of graphs. The transi- 
tion matrix q is here arbitrary. 

Definition 8 Let C be a proper subset of S, and let rj > 0. A graph g £ 
G(C) is 77-maximal if 

p(q) > 77 max p(q'). 

We denote by G V (C) the set of 77-maximal C-graphs. For simplicity of 
notations, we do not emphasize the dependency of G V (C) on the transition 
matrix. Clearly, G V (C) is non-empty, for every 77 < 1 and C C S. It is 
worth listing a few basic properties of graphs that we use repeatedly. 

Proposition 9 PO Let C 1 nC 2 = 0, and gi £ G{Ci), for i = 1, 2. // all 



paths of gi lead to C\ U C2, then g\ U g 2 is a Ci U C 2 - graph. 
PI Let Ci n C 2 = 0, g € C(Ci U C 2 ), and gi the restriction of g to Ci. If 



all paths of g 2 lead to C\ U C 2 , then g\ € G' ? (Ci). 
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P2 Let dr\C 2 = 0, and g, L G G* (C<) /or i = 1,2. // #1 U 52 is a Ci U C 2 - 
graph, then it is 771772 -maximal. 



Proof. PO and P2 follow from the definitions. We now show that PI 
holds. Otherwise, there is g[ G G(C±) such that p{gi) < r)p(g'i). By PO, 
g' = g[ U g 2 is in G(C\ U C2), but p(g) < rjp(g'), a contradiction. ■ 

Note that PI needs not hold without the condition that all paths of g 2 



lead to d UC 2 . Indeed, take S = {1,2,3,4}, C x = {1}, C 2 = {2}, and 
g(2 I 1) = g(l I 2) = 1 - q(3 | 1) = 1 - g(4 | 2) = 2/3. The C 2 -graph 
5l = (2 — > 4) is 1/2-maximal, and the C\ U C 2 -graph (1 — > 2, 2 — > 4) is 
1-maximal. 

Lemma 10 Let C be a proper subset of S, let 77 > 0, and let H be a set of 
graphs such that G ri (C) C H C G(C). T/ien 

0<%g^-l<7 ? L. 



In particular, 



< 1 - p-r < r?L. 



Proof. Since H C G(C), and by the definition of 6F»(C), 

< E g gG(c)P(g) _ j = J2 9 eG(c)\H p(g) < E g gG(o\G"(c)P(g) < ^ L 

as desired. ■ 



4 Proof of the main results 

We here prove Theorems |2] and El We let e, (3 G (0, 1) satisfy the assumptions 
of Theorem 121 and gbe another transition matrix over S. We assume that 
<f is (e, /3)-close to 
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4.1 On graphs 

For every proper subset C of S and every 77 > 0, we denote by G n (C) the set 
of ?7-maximal graphs under q. For every C-graph g, p(g) = Y\ seC q(g(s) \ s) 
is the weight of g under q. 

Lemma 11 For every proper subset C of S, 

Y, < ^q{C\s) < (1 + P) \S\ 2 Y, Vsq(C\s). (9) 

' ' s&C s&C s&C 

Proof. Let sq G C and to & C maximize the quantity fJ, s q(t \ s) amongst 
s G C and t G C. Then fi so q(t \ s ) > EseC ^q(C\s)/\S\ 2 > C q /\S\ 2 > e( q . 
Since q is (e,/3)-close to q, q(to \ sq) > (1 — 0)q(to \ sq). In particular, 



Y^qiC I s) > HsoWo I s o) > (1 - /3)/i ao g(to I s ) > \^r^ ^ Ai s g(C|s 
sec ' ' sec 

(10) 



and the left hand side inequality in @ holds. 

Let si G C and t\ £ C maximize the quantity /J, s q(t \ s) amongst s G C 
and t EC. By <©, /i si ?(ii | a x ) > E 86 c^ I *)/|S| 2 > (1 - /3)C g /|S| 4 > 
e. Since g is (e, /3) -close to g, | si) > <f(*i | si)/(l +/3). Therefore 

2 1 

^J/x s g(C [ s) > /i si g(ti I si) > I fi i) > (! + < g)|g|2 X^ S ^(C ' s )' 

and the right hand side inequality holds as well. ■ 

Lemma 12 Let C C S and s G C be given. For every g G G^{C) (resp. 
g G GP(C)) fj> s q(g(s) \ s) > e( q (resp. n s q(g(s) \ s) > eQ). 

Note that the second claim is not symmetric to the first, since in both 
we use the stationary distribution of q. 

Proof. The proof is quite similar for g G G@{C) and g G G^{C). We 
prove the lemma for the former, and mention where the proof for the latter 
differs. 

Let g G G@(C) be arbitrary. The proof is by induction over the number 
of states in C. 
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If |C| = 1, then C = {s} for some s G S. Since g is /^-maximal, (J- S q(g(s) \ 
s) > (3/\S\n s q(C\s) > fap (for g G G^C), by Lemma EU /i s g( 5 (s) | s) > 
llyM^ls) > P^fi s q(C\ S ) > P^rQ- 

Consider now the case \C\ > 1. 

We first assume that there are at least two edges of g whose endpoints 
do not belong to C. Let s\ ^ s 2 G C Let gi be the restriction of g to C\{sj}, 
i = l,2. By PI, gi G G@{C \ {si}). Since any edge of g is an edge of g\ or 
g2 (or both), the induction hypothesis applied to C \ {s«} and gi, i = 1,2, 
implies that the claim holds for g. 

Assume now that there is a unique state s\ G C such that g{s\) C. 
Let g 1 be the restriction of g to C \ {«i}. By PI, 51 G (^(C \ {s x }). By 
the induction hypothesis applied to C \ {s\} and g±, fi s q(g(s) \ s) > eC q for 
every s G C \ {s\}. Thus, it remains to show that fi Sl q(g(si) | si) > e( q . 

Let S2 G C maximize the quantity fJ. s q(C \ s) amongst s G C (for g G 
G@{C), it is chosen to maximize ^ s q(C [ s)). By the definition of ( q , [i S2 q(C j 
82) > CJ\S\ (for g G G^C), by LemmaUB f, S2 q(C \ s 2 ) > (1 - /?)C 9 /|5| 3 ). 
Let 5 G G 1 (S\C) (for 5 G C^(C), one also chooses 5 G G 1 (S\C)) . By PO 
and P2, 5U 5l GG(S\{ Sl }). 

Let 5 G G 1 (5' \ {s 2 })- Since is a S \ C-graph, we have p(g) > 

P(~9\S\c)- Since for ever y * e #|C\{s 2 } U ( s 2'0 is a C'-graph, p(g) > 
p(<?|C\{s 2 })<?(* I s 2 )- In particular, p{g)p{g) > Pp(g)q{t | s 2 ) for every f G C, 
and therefore p(g)p(g) > jLp(g)q(C | s 2 ). 
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Denote £ = ~£ y es E 9 eG(s\{y}) via )■ By ®, ^ = j* E g eG(S\{ s }) Pis)- 
In particular, 

< Vs 2 qlC I s 2 ) < g x q{C | s 2 ) 

Lp(g)g(C | s 2 ) Lx\S\ 

< g < ^ P{9)P{9) 

L x ISI 

-p(gUg\ (s 1 ,g(sx)))q(g(s 1 ) | s x ) 



£0 

x |5 

(j> si q(g(si) I si). 



geG(S\{ Sl }) 

L x 15 



But then ^ S iQ(g{si) | si) > eCg> as desired. The calculation for g £ G@(C) 
is analogous. ■ 

Corollary 13 For every proper subset C of S, 

p(g) 



. < (|5| + for every g € G^(C) U G^C) (11) 
and 

< (\S\ + 1)13, where H = G P (C) U G P {C). 



E g eHP(g) _ 

J2 g& HP(g) 



Thus, the weights of /^-maximal graphs under q and q are close. 

Proof. Note first that the second inequality follows immediately from 
the first one. Let us prove (fTTj). Let g G G p [C). By Lemma H n s q{g{s) \ 
s) > £(q for every s G C. Since q is (e,/3)-close to g, (1 — (3)q(g(s) \ s) < 
q(g(s) | s) < (1 + P)q(g(s) | s). Multiplying this inequality over s £ C yields 
(1 - (3)\°\p{g) < p(g) < (1 + P) lcl p(g), and (|TTJ) follows. 

The proof for g € G^(C) is similar. ■ 

4.2 Proof of Theorem [U 

Proposition 14 T/ie transition matrix q is irreducible. 
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Proof. It is enough to prove that for every non-empty subset C C S, 
there exists s G C, and t £ C such that q(t \ s) > 0. 

Let s± G C and t\ C be such that /j, si q(ti \ s\) > ( q /\S\ 2 > eC, q . Since 
g is (e,/3)-close to g(ii | s\) > (1 — (3)q(t \ s) > 0. ■ 

We need the following technical Lemma. 



Lemma 15 1. Let (aj)f =1 and {bi)l =1 be positive numbers, and let e > 



0. // 



9± _ 1 
bi 1 



min{ai,a2,...,a/} _ 
min{6i,fe2,-ib/} 



< e for every i = 1, . . . , I then 
< e. 



Ei=l b i 



< e and 



2. Let e G (0,1/3), and let a,A,b,B > 0. // | f - l| < e and \ ^ - l| < e 



then 



a/b 
A/B 



1 



< 3e. 



Proof. The proof of the first part is left to the reader. For the second 
part, note that 1/(1 + e) < B/b < 1/(1 — e), which implies that B/b — 1 < 
e/(l — e). In particular, 



a/b 



A/B 



< 



i- 1 



+ 1 



B 

j- 1 



+ 



a 

A' 1 



< (1 + £)- — - +£ < 3e. 



Proposition 16 For each s G 5, 

|1 - 

Proof. Fix s G S. By ®, 

Ec?(5\{ s })P(ff) 



Ms 

Ms 



< 18/3L. 



Ms 



and /I s 



E 



G(S\{s}) 



Ej/eS Eg(S\{j/}) EyG5 Eg(S\{j/}) P(d) 

For every y G S, define ff y = G^(S\ {y}) U G^(5\ {y}). Define 



Eg, Kg) 
Ej,g5 Eh w p(s) 



and Ms 



Eg s F(g) 
E^S Eh„ ' 



By Lemma EH an d Lemma 1151 



Ms _ 1 

5 



< 3/3L and 



< 3/3L. By 



Lemmas and 1151 
follows by Lemma EMU- 



_ 1 

7s 1 



< 3(151 + l)/3. Since L > |5| > 2, the result 
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4.3 Proof of Theorem 03 

Proposition 17 For every proper subset C of S, every s S C and t ^ C, 

\Q s , q (t \ C) - Q St g(t \ C)\ < 12(3L. 

Proof. Denote H = G^{C) U Gfl(C), and H s>t = H n G Sjt (C). 
Assume first that H S;t ^ 0. By Q, one has 



< PL. 



Since 



Q a ,,(*|C) 



< this yields, by Lemma IToT|2*)) . 



EffP(s) 



< /3L + 3/3L < 4/3L, 



and a similar inequality holds with g replaced by g. 
By Corollary fTTA and Lemma IT5lfT]) . 



E#nG s ,t(G)P(sO 



< (3(\S\ + 1) and 



EhP(s) 



</?(|5| + l) 



By Lemma EMU) C3) implies 

E/fnG s , t (G) E//nG s , t (C) P(s0 



EffKs) 
which implies, using (fT2*|) . 



<3/3(|5| + l), 



|Q s , 3 (t|C) - Q a , ? (t|C)| < /?(8L + 3(|5| + 1)). 



(12) 



(13) 



If, on the other hand, H St t 
Q., q (t | C),Q Sj? (t | C) </?L. I 



then by ® and the definition of H s t, 



Proposition 18 For every proper subset C of S, 

1 



2151 



T K C < K c <2\S\ Z K C . 
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Proof. We first argue that 

Kc = v E ' eC ^, y (14) 

Indeed, define the r.v. p n as the average length of visits to C that end before 
stage n + 1: 

zJ P =i 1 s p ec 

Pn = — : : • (15) 

2^p=l L s p €C L s p+1 (£C' 

By the ergodic theorem, the sequence (p n ) converges, P Sli9 -a.s. to Kc, while 
the right hand side in (TT51) converges P Sl -a.s. to ^ s6C ^1 . The identity 

(HU) follows. 

By Proposition El for every s € C, 



(1 - 18/?L)/i s <&<(! + 18/3L)/i s . (16) 



By Lemma ITTl 

1-/3 



1 1 sec sec sec 

Eqs. (HSJ and |T7|) yield 

(1 ~ 18 S (1 ~ /?) E^l g ) ^ E^^ s ) ^ (l+/3)(l+18/3i)|^| 2 E^k) 

sec sec sec 

(18) 

Summing up Eq. ((TO)) over s € C gives 

(i - 18/3L) »s < E & < C 1 + 18 ^ L ) E ( 19 ) 

sec see sec 
The Proposition follows by dividing (fTT?|) by l|18j). ■ 



5 Proof of the variations 

We here prove Theorems 0] and |H1 We shall follow the previous proofs, 
and will point out which changes are needed. We let o, e, j3 be given, that 
satisfy the assumptions of Theorem 2] The result of Section T4. II still hold for 
every proper subset C oi Si, namely Lemmas ^2 ^] and Corollary are 
still valid, provided the assumption C C S is replaced by the assumption 
C c Si. 
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5.1 Proof of Theorem |U 

We need the following observation. 

Lemma 19 For every y G Si, there exists a (a/ Liy s \ -maximal graph ~g G 
G(Si\y) such that all paths ofg lead to y. 

Proof. By P2, for every s G Si\y there is a ^-maximal £i\y-graph 
g s in which s leads to a state in y. Let h s be the path in g s that connects 
s to y (this is a set of edges). Let ~g be a Si\y-graph that is contained in 
Ug^SxXyhs- Then 5 satisfies the conditions. ■ 

We next prove the two assertions of Theorem HJ 

Lemma 20 All states of Si belong to the same recurrent set for q. 

Proof. It is enough to prove that for each C C Si, there exists s G C 
and t G C such that q(t\s) > 0. The proof of Proposition ITU still applies, 
provided ( g is replaced by Q. ■ 



Lemma 21 For each s £ Si, 



fi(s\Si 



< 180L. 



fJ,(s\Si) 

Proof. The proof goes essentially as in Proposition^! Set rj = (3/(a/L) I 5 ' < 
(a/L) s , and fix s G Si. By ©, 

E G (S\{s})P(9) , _ , e . _ E G (S\{s})P(9) 



fJ,(s\Si) 



and /J(s|5i 



EyeSi T,G(S\{y}) ^(f ) E y eSi Eg(s\M) 

For every y € Si, define # y = GP(S\ {y}) U G^(5\ {y}). Define 



A*'(s|5i) 



and ju / (s|5'i) 



Eh s 



Eygs! Eh„ p{g) EyeSi E^ 

Fix for a moment y £ Si. By Lemma H^l there is a (a/L) s l -maximal Si\{y}- 
graph g such that all its paths lead to y. Let g G G n (S\{y}), and Ss^Uj/}, 
<7s\Si its restrictions to Si\{y} and S\Si. Using the above remark, the 
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graph gU gs\Si 1S a S\ {y}-graph. Therefore, 9si\{y} is 77(a/L)' s '-maximal 



(= /3-maximal). By Corollary El one has 



P(9Sl\{i/}) 



< (\S\ + l)/9. 



Since q and <f coincide outside Si, p(gs\Si) = P(ds\Si)- Thus, 
(\S\ + LemmaEl an d Lemma El implies that 



p(g) 
TO 



< 



Ms\Sl) 



< 3/3L. By Lemmas [TO] and El 



Si) j 



1 



< 3/3 L and 



3TJ 



< 3(|5| + 



Since L > > 2, the Lemma follows by Lemma EMU)- ■ 
5.2 Proof of Theorem [U 

The proof of the first two assertions in Theorem H3 is identical to the proof 
of the two assertions in Theorem [5] (see Propositions 1171 and I18j) . We omit 
it. We now prove a slightly strengthened version of the last assertion. 



Proposition 22 Let rj < eQ be such that 
rj. One has 

1 



1 



EM 

q(t\s) 



< (3 whenever fj, s max(q(t\s),q(t\s)) > 



1 



-K Sl < K Sl < cK Sl or K Sl , K Sl > 
c 2d 



7] 



The last statement in Theorem corresponds to the case rj 
Proof. Fix s £ Si. By (UU), 

K Sl ' 



EteSi Kt\Si)q(Si | t) 
and a similar equality holds for Ks x , involving /2 and q. By Theorem 01j2J) 
and Lemma El the ratio between K$ 1 and — j is between 



1 - 54/5L and 1 + 54/3L. 

If for every t & Si and u ^ Si, p-tq(u \ t) < r\ and [itqiu \ t) < rj, then 
K Sl > ^ and K Sl > (1 - 54/3L) x as desired. 



If, on the other hand, there exist t € Si and u Si such that jitq{u \ t) > 
77 or Htq(u I i) > 77 then |1 — | < /?, and therefore jitq{u \ t) > (1-/3)7/ and 
lHq(u \ t) > (1 — f3)r). For every t & Si and u ^ Si such that fMq(u \ t) < rj 
and Htq(u \ t) < rj we have jitq{u \ t) < X^eSi ^tQ^Si \ t) and jitq(u I i) < 
^teSi ^tk\Si \t). It follows that the ratio between X^gSj /^(^i I anc ^ 
Sig5! fHq(Si I t) is at most IS 1 ) 2 . The result follows. ■ 
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